FINAL  REPORT 


MUNITIONS  CLASSIFICATION  WITH  PORTABLE  ADVANCED 
ELECTROMAGNETIC  SENSORS 

Demonstration  at  the  former  Camp  Beale,  CA,  Summer  201 1 


JULY  2012 


Herbert  Nelson 
Anne  Andrews 

SERDP and  ESTCP 


This  document  has  been  cleared  for  public  release 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

JUL  2012 

2.  REPORT  TYPE 

N/A 

3.  DATES  COVERED 

4.  TITLE  AND  SUBTITLE 

5a.  CONTRACT  NUMBER 

Munitions  Classification  With  Portable  Advanced  Electromagnetic 

Sensors  Demonstration  at  the  former  Camp  Beale,  CA,  Summer  2011 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

SERDP;  ESTCP  Office  4800  Mark  Center  Drive  Suite  17D08  Alexandria, 
VA  22350-3605 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR'S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release,  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 
ABSTRACT 

SAR 

18.  NUMBER 

OF  PAGES 

46 

19a.  NAME  OF 
RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


TABLE  OF  CONTENTS 


Figures . v 

Tables . vii 

Acronyms . viii 

Acknowledgements . ix 

Executive  Summary . ES-1 

1  Introduction . 1 

1.1  B  ackground . 1 

1.2  Classification  Concept . 2 

1.3  Results  from  the  MetalMapper  Demonstration  at  Camp  Beale . 3 

1.4  About  this  Report . 4 

2  Former  Camp  Beale  Demonstration  Site . 5 

2. 1  Site  History  and  Characteristics . 5 

3  Program  Design . 8 

3 . 1  Overall  Approach . 8 

3.2  Demonstration  Preparation . 9 

3.2.1  Seeding  the  Site . 9 

3.2.2  Instmment  Verification  Strip  and  Training . 9 

3.2.3  Surface  Clearance . 10 

3.3  Targets  of  Interest . 10 

3.4  Data  Collection . 11 

3.5  Classification  Approaches . 11 

3.5.1  Geophysical  Models . 11 

3.5.2  Classifiers . 12 

3.6  Classification  Product . 13 

3.7  Scoring  Methods . 14 

4  advanced  Sensor  technologies . 15 

4.1  Man-portable  Vector  Sensor  (MPV) . 15 

4.2  TEMTADS  2x2 . 16 

4.3  Berkeley  UXO  Discriminator  (BUD) . 17 

5  Anomaly  Selection  and  Investigation  results . 19 

5.1  Anomaly  Selection . 19 

5.1.1  Anomaly  Selection  Threshold . 19 

iii 


5.2  Geolocation  Accuracy . 20 

5.3  Intrusive  Investigation . 21 

6  Performance  of  the  Portable  Sensors . 23 

7  Classification  Results . 24 

7. 1  EM6 1 -MK2  CART . 24 

7.2  MPV . 25 

7.3  TEMTADS  2x2  Array . 27 

7.4  Portable  BUD . 28 

7.5  Results  Overview . 29 

8  Cost  Savings . 30 

9  Conclusions . 32 

10  References . 33 


FIGURES 


Figure  1-1.  Results  of  SAIC  analysis  of  MetalMapper  data  acquired  by  Parsons  at  the  former 

Camp  Beale . 4 

Figure  2-1.  Aerial  photograph  of  the  former  Camp  Beale  FUDS  with  historic  ranges  overlain. 

The  50-acre  site  boundary  which  contains  the  demonstration  areas  is  shown  in  the 
blow-up . 6 

Figure  2-2.  The  fifty-acre  demonstration  site  with  the  MetalMapper,  portable  system,  and 

combined  areas  delineated . 7 

Figure  2-3.  Measured  geologic  background  variability  in  the  Combined  Grids  of  the  former 

Camp  Beale  site . 7 

Figure  3-1.  Flow  chart  outlining  steps  in  the  demonstration  at  Camp  Beale.  Blue  boxes  are  tasks 

performed  by  ESTCP.  Others  are  tasks  performed  by  contractors . 8 

Figure  3-2.  Examples  of  the  items  removed  during  the  pre-survey  surface  clearance . 10 

Figure  3-3.  Two  "empty"  37-mm  projectiles  recovered  in  this  demonstration . 11 

Figure  3-4.  Format  for  the  initial  (left)  and  final  (right)  ranked  anomaly  lists  for  this 

demonstration . 13 

Figure  3-5.  Example  receiver  operating  characteristic  curve . 14 

Figure  4-1.  Components  of  the  MPV.  Left  inset  shows  data  acquisition  (DAQ)  and  power  unit 
mounted  on  a  backpack  frame.  Right  panel  shows  view  of  sensor  head  from  above 
with  cube  numbers . 15 

Figure  4-2.  MPV  data  collection  in  the  trees  at  Camp  Beale . 16 

Figure  4-3.  Schematic  of  the  TEMTADS  2x2  array . 16 

Figure  4-4.  Portable  TEMTADS  2x2  data  acquisition  at  Beale . 17 

Figure  4-5.  Transmit  and  receive  coil  configuration  for  the  prototype  portable  UXO 

discriminator . 17 

Figure  4-6.  Portable  Berkeley  UXO  Discriminator  at  Camp  Beale . 18 

Figure  5-1.  Predicted  EM61-MK2  anomaly  amplitude  in  gate  2  for  a  37-mm  projectile  in  its  least 
and  most  favorable  orientations.  Also  shown  are  the  RMS  noise  measured  at  the 
site,  the  30  cm  depth  used  to  set  the  threshold  and  the  anomaly  selection  threshold 
used  in  this  demonstration . 19 


v 


Figure  5-2.  Comparison  of  the  distance  between  seed  location  and  nearest  anomaly  location  in 
the  EM61  survey  data  from  the  treed  area  at  Camp  Beale  (left)  and  Camp  Butner 
(right) . 20 

Figure  5-3.  Examples  of  the  clutter  items  recovered  in  this  demonstration . 21 

Figure  5-4.  Histogram  of  recovery  depths  for  all  items  other  than  seeds . 22 

Figure  6-1.  Polarizabilities  for  four  items  from  the  IVS  derived  from  measurements  of  the  three 
portable  sensors  used  in  this  demonstration  compared  to  those  derived  using  the 
MetalMapper . 23 

Figure  7-1.  SAIC  analysis  of  the  EM61-MK2  cart  data  using  UX-Analyze . 24 

Figure  7-2.  ROC  curve  resulting  from  Sky's  statistical  classifier  applied  to  the  EM61-MK2  cart 

data . 25 

Figure  7-3.  ROC  curve  resulting  from  the  Sky  analysis  of  MPV  cued  data  collected  at  Camp 

Beale . 26 

Figure  7-4.  ROC  curve  resulting  from  the  SAIC  analysis  of  MPV  cued  data  collected  at  Camp 

Beale . 26 

Figure  7-5.  ROC  curve  resulting  from  the  Dartmouth  analysis  of  MPV  cued  data  collected  at 

Camp  Beale . 27 

Figure  7-6.  Sky  analysis  of  the  TEMTADS  2x2  cued  data  collected  at  Camp  Beale . 27 

Figure  7-7.  SAIC  analysis  of  the  TEMTADS  2x2  cued  data  collected  at  Camp  Beale . 28 

Figure  7-8.  Sky  library-match  analysis  of  the  BUD  cued  data . 28 

Figure  7-9.  Sky  analysis  of  BUD  cued  data  using  a  statistical  classifier . 29 

Figure  7-10.  Overview  of  the  performance  of  all  analyses  of  the  cued  data  from  the  advanced 


portable  sensors  at  Camp  Beale.  All  sixteen  analysts  correctly  identified  100%  of  the 
TOI.  The  bars  represent  the  number  of  clutter  items  dug  at  the  analysts  chosen 
threshold.  The  orange  bars  show  the  number  of  clutter  that  would  have  to  be  dug  if 
the  analyst  was  able  to  put  their  threshold  in  the  optimum  place.  This  represents  the 
best  performance  possible  from  each  sensor/ analysis  combination . 29 


vi 


TABLES 


Table  1-1.  Participants  in  the  Portable  Sensor  Demonstration  at  the  former  Camp  Beale . 2 

Table  3-1.  Seeds  Emplaced  for  the  Camp  Beale  demonstration  (includes  the  MetalMapper 

demonstration  area) . 9 

Table  3-2.  Details  of  the  Instrument  Verification  Strip . 10 

Table  5-1.  Depths  to  which  TOI  on  Camp  Beale  will  be  detected  at  a  threshold  of  5.2  mV . 20 

Table  5-2.  Distribution  of  recovered  items  from  this  demonstration . 21 

Table  6-1.  Average  daily  productivity  for  the  portable  sensor  systems . 23 

Table  8-1.  Unit  cost  assumptions . 30 

Table  8-2.  Cost  Comparison  for  100  acres  of  a  site  comparable  to  Camp  Beale . 31 


vii 


ACRONYMS 


BUD 

Berkeley  UXO  Discriminator 

DoD 

Department  of  Defense 

EM  (I) 

Electromagnetic  (Induction) 

ESTCP 

Environmental  Security  Technology  Certification  Program 

FUDS 

Formerly  Used  Defense  Site 

GPS 

Global  Positioning  System 

IDA 

Institute  for  Defense  Analyses 

IMU 

Inertial  Measurement  Unit 

IVS 

Instmment  Verification  Strip 

MM 

MetalMapper 

MMRP 

Military  Munitions  Response  Program 

MPV 

Man-portable  Vector  Sensor 

MR 

Munitions  Response 

MTADS 

Multi-sensor  Towed  Array  Detection  System 

NRL 

Naval  Research  Laboratory 

QC 

Quality  Control 

ROC 

Receiver  Operating  Characteristic 

SERDP 

Strategic  Environmental  Research  and  Development  Program 

SLO 

San  Luis  Obispo 

SNR 

Signal  to  Noise  Ratio 

TEMTADS 

Time  Domain  Electromagnetic  MTADS 

TOI 

Target  of  Interest 

UXO 

Unexploded  Ordnance 

viii 


ACKNOWLEDGEMENTS 


This  demonstration  in  the  ESTCP  Classification  Pilot  Program  would  not  have  been  possible 
without  the  assistance  of  numerous  individuals.  We  would  like  to  acknowledge  Shelley  Cazares, 

Mike  Tuley,  and  Elizabeth  Ayers  from  the  Institute  for  Defense  Analyses  who  were  instmmental  in 
the  overall  program  design  and  analysis  of  the  demonstrator  results.  We  would  also  like  to  thank  the 
ESTCP  Classification  Study  Advisory  Group  listed  below.  They  were  involved  in  site  selection, 
program  design,  data  review,  and  the  development  of  conclusions  and  methods  as  a  whole. 

Jim  Austreng,  U.S.  Army  Corps  of  Engineers,  Sacramento  District 

Harry  Craig,  U.S.  EPA  Region  10,  Oregon  Operations  Office 

Bryan  Harre,  Naval  Facilities  Engineering  Service  Center,  Port  Hueneme,  CA 

Robert  Kirgan,  Army  Environmental  Command 

Doug  Maddox,  US  Environmental  Protection  Agency 

Doug  Murray,  Naval  Ordnance  Safety  and  Security  Activity 

Andrew  Schwartz,  U.S.  Army  Corps  of  Engineers,  Huntsville 

Stephen  Sterling,  California  Department  of  Toxic  Substances  Control 

Jeff  Swanson,  Colorado  Department  of  Public  Health  and  Environment 

Ken  Vogler,  Colorado  Department  of  Public  Health  and  Environment 

Ed  Walker,  California  Department  of  Toxic  Substances  Control 

Amy  Walker,  U.S.  Army  Corps  of  Engineers,  Huntsville 

Dan  Ward,  California  Department  of  Toxic  Substances  Control 

Roger  Young,  U.S.  Army  Corps  of  Engineers,  Huntsville  (retired) 

We  would  like  to  credit  the  technology  demonstrators. 

•  Dartmouth,  Lawrence  Berkeley  National  Laboratory,  SAIC,  Signal  Innovations  Group,  and 
Sky  Research  for  classification  data  analysis; 

•  Sky  Research  and  G&G  Sciences,  Nova  Research,  and  Lawrence  Berkeley  National 
Laboratory,  for  data  collection  for  MPV,  TEMTADS  2x2,  and  the  portable  BUD 
respectively. 

For  program  support,  we  acknowledge 

•  Parsons  for  seeding,  EM61-MK2  data  collection,  and  validation  excavations; 

Finally,  ESTCP  thanks  Mr.  Tim  Caldwell  and  Mr.  Mark  Carroll  of  the  California  Department  of  Fish 
&  Game  for  their  patience  and  cooperation  during  the  demonstration. 


IX 


X 


EXECUTIVE  SUMMARY 


Classification  using  portable  advanced  electromagnetic  sensors,  designed  to  operate  in  areas  where 
terrain  and  vegetation  preclude  the  use  of  vehicle-borne  sensors,  was  demonstrated  at  the  former 
Camp  Beale,  CA  in  2011.  The  TEMTADS  2x2,  Man-Portable  Vector  (MPV)  and  portable  Berkeley 
UXO  Discriminator  (BUD)  were  used  to  successfully  classify  all  of  the  targets  of  interest  by  all 
analysts.  There  was  some  variation  among  analysts  in  the  percent  of  clutter  rejected.  All  but  two  of 
the  1 6  performers  eliminated  about  7 5%  of  the  clutter.  This  is  comparable  to  the  results  achieved 
using  the  vehicular-borne  MetalMapper  on  another  part  of  this  site. 

Even  though  this  was  the  first  live  site  demonstration  for  each  of  the  sensors,  daily  productivity  of 
90  to  175  anomalies  was  possible.  One  of  the  teams  was  able  to  collect  cued  data,  extract 
parameters,  and  classify  for  $26  per  anomaly.  Using  this  per  anomaly  cost  for  classification  and  a 
few  reasonable  assumptions  we  calculate  that  the  use  of  classification  would  result  in  a  50%  savings 
for  a  100-acre  remediation  on  a  site  with  conditions  like  Camp  Beale. 

When  a  site  is  remediated,  it  is  typically  mapped  with  a  geophysical  system,  based  on  either  a 
magnetometer  or  electromagnetic  induction  (EMI)  sensor,  and  the  locations  of  all  detectable  signals 
are  excavated.  Many  of  these  detections  do  not  correspond  to  munitions,  but  rather  to  other 
harmless  metallic  objects  or  geology:  field  experience  indicates  that  often  in  excess  of  99%  of  objects 
excavated  during  the  course  of  a  munitions  response  are  found  to  be  nonhazardous  items.  As  a 
result,  most  of  the  costs  to  remediate  a  munitions-contaminated  site  are  currently  spent  on 
excavating  targets  that  pose  no  threat.  If  these  items  could  be  determined  with  high  confidence  to 
be  nonhazardous,  some  of  this  expense  could  be  avoided  and  the  available  funding  applied  to  more 
sites. 

Classification  is  a  process  used  to  make  a  decision  about  the  likely  origin  of  a  signal.  In  the  case  of 
munitions  response,  high-quality  geophysical  data  can  be  interpreted  with  physics-based  models  to 
estimate  parameters  that  are  related  to  the  physical  attributes  of  the  object  that  resulted  in  the  signal, 
such  as  its  physical  size,  aspect  ratio,  wall  thickness,  and  material  properties.  The  values  of  these 
parameters  may  then  be  used  to  estimate  the  likelihood  that  the  signal  arose  from  an  item  of  interest, 
that  is,  a  munition. 

The  Environmental  Security  Technology  Certification  Program  (ESTCP)  is  charged  with 
demonstrating  and  validating  innovative,  cost-effective  environmental  technologies.  ESTCP 
recently  initiated  a  Classification  Pilot  Program,  consisting  of  demonstrations  at  a  number  of  sites,  to 
validate  the  application  of  a  number  of  recently  developed  technologies  in  a  comprehensive 
approach  to  munitions  response. 

The  goal  of  the  pilot  program  is  to  demonstrate  that  classification  decisions  can  be  made  explicitly, 
based  on  principled  physics-based  analysis  that  is  transparent  and  reproducible.  As  such,  the 
objectives  of  the  pilot  program  are  to: 

•  test  and  validate  detection  and  classification  capabilities  of  currently  available  and  emerging 
technologies  on  a  real  site  under  operational  conditions,  and 

•  investigate  how  classification  technologies  can  be  implemented  in  cleanup  operations  in 
cooperation  with  regulators  and  program  managers. 


ES-1 


ES-2 
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INTRODUCTION 


1.1  BACKGROUND 

Classification  using  portable  advanced  electromagnetic 
sensors  was  demonstrated  at  the  former  Camp  Beale,  CA  in 
2011.  The  TEMTADS  2x2,  Man-Portable  Vector  (MPV) 
and  portable  Berkeley  UXO  Discriminator  (BUD)  all 
collected  data  to  support  successful  classification.  Camp 
Beale  was  also  the  site  of  a  demonstration  of  the 
MetalMapper  advanced  sensor,  described  in  a  separate 
report.  (Ref.  1) 

Classification  is  motivated  by  the  need  to  perform 
munitions  response  more  cost-effectively  so  that  limited 
clean  up  dollars  can  be  used  to  reduce  real  risk  on 
munitions-contaminated  sites  sooner.  The  estimated 
liability  in  the  FY 1 0  Defense  Environmental  Programs 
Report  to  Congress  for  Munitions  Response  is  $15. 2B.  (Ref. 

2)  The  bulk  of  this  liability  is  $10.0B  for  the  1703  sites 
identified  in  the  Formerly  Used  Defense  Sites  (FUDS) 
program  and  $4.4B  for  the  2433  sites  identified  on  Active 
Installations.  The  remaining  $0.8B  is  in  Base  Realignment 
and  Closure  (BRAC).  The  estimated  completion  dates  for 
many  sites,  particularly  in  the  FUDS  program,  are  decades 

using  current  practice. 

When  a  munitions  response  site  is  cleaned  up,  in  most 
cases,  it  is  mapped  with  a  geophysical  sensor  and  the 
locations  of  all  detectable  signals  are  excavated. 

Geophysical  sensors  detect  metal  and,  therefore,  many  of  the  detections  do  not  correspond  to 
munitions,  but  rather  to  harmless  metallic  objects.  Field  experience  indicates  that  95-99%  or  more  of 
objects  are  found  to  be  nonhazardous.  Current  technology  does  not  provide  a  means  to 
discriminate  between  munitions  and  other  items,  termed  “clutter.”  As  a  result,  most  of  the  costs  to 
remediate  a  munitions-contaminated  site  using  current  methods  are  spent  on  excavating  targets  that 
pose  no  threat.  If  buried  items  could  be  reliably  classified  into  those  that  are  munitions  and  those 
that  are  not,  only  the  munitions  would  need  to  be  removed  from  the  site. 

The  Strategic  Environmental  Research  and  Development  Program  (SERDP)  and  the  Environmental 
Security  Technology  Certification  Program  (ESTCP)  have  supported  the  development  of  purpose- 
built  advanced  electromagnetic  sensors  and  associated  analysis  methods  for  classification.  Following 
the  successful  demonstration  of  classification  methods  in  controlled  test  environments,  ESTCP 
initiated  a  Classification  Pilot  Program  to  validate  the  application  in  real-world  conditions.  The 
demonstrations  are  planned  and  conducted  in  cooperation  with  regulators  and  program  managers  in 
the  Services. 


out  if  they  are  to  be  cleaned  up  at  planned  funding  levels 


Former  Camp  Beale,  CA  -  complex 
historical  usage,  overlapping  network  of 
ranges  throughout,  moderate  geologic 
interference.  The  portable  demonstration 
area  had  moderate  slopes  and  tree  cover. 


Munitions  —  37-mm projectiles,  60-mm 
mortars,  81  -mm  mortars,  105 -mm 
projectiles 


Results  -  2X2  TEMTADS,  MPV 
and  portable  BUD  were  used  to 
successfully  classify  all  of  the  targets  of 
interest  by  all  analysts.  There  was  some 
variation  among  analysts  in  the  percent 
of  clutter  rejected.  All  but  two  of  the  1 6 
pe  formers  eliminated  about  75%  of  the 
clutter.  This  is  comparable  to  the  results 
achieved  using  the  MetalMapper  on  this 
site. 


1 


The  goal  of  the  program  is  to  demonstrate  that  classification  decisions  can  be  made  using  an  explicit 
approach,  based  on  principled  analysis  that  is  transparent  and  reproducible.  Demonstrations  will  be 
required  at  a  number  of  sites  to  represent  the  wide  variability  in  munitions  types,  target  densities, 
terrain,  vegetation,  geology,  land  use  history,  future  land  use,  and  other  site  characteristics  that  will 
affect  the  applicability  of  classification  and  to  establish  cost  effectiveness  and  implementability.  The 
demonstrations  also  present  an  opportunity  to  work  out  standard  operating  procedures  and  establish 
quality  control  (QC)  measures.  Details  about  past  and  ongoing  demonstrations  can  be  found  on  the 
SERDP-ESTCP  web  site  at  http:/ / serdp-estcp.org/Featured-Initiatives/Munitions-Response- 
Initiatives/  Classification- Applied-to-Munitions-Response. 

The  MetalMapper  and  full-sized  TEMTADS  and  BUD  have  been  successfully  demonstrated  in  prior 
live-site  efforts.  These  are  large,  heavy  sensors  suitable  for  use  in  open  areas  where  they  can  be 
maneuvered  with  vehicles.  The  demonstration  at  the  former  Camp  Beale  is  the  first  test  of  smaller 
man-portable  advanced  EM  sensors  intended  for  use  where  terrain  or  vegetation  demand  a  more 
maneuverable  option.  They  are  less  mature  and  field  procedures  were  not  established  prior  to  this 
demonstration.  Data  were  analyzed  by  experienced  teams  from  the  developers  of  the  classification 
methods.  These  are  important  considerations  in  evaluating  and  applying  the  results,  particularly  the 
production  rates.  Production  rates  will  likely  increase  with  use  and  analysis  methods  will  transition 
to  production  geophysicists,  although  they  have  not  yet.  For  these  reasons,  we  discourage  potential 
customers  from  using  the  results  from  this  demonstration  to  rank  performers  and  make  contracting 
selections.  As  we  conduct  more  demonstrations  of  the  portable  sensors  on  a  variety  of  sites,  the 
identities  of  the  consistently  better  performers  will  emerge. 

Table  1-1  shows  the  participants  and  their  roles  in  the  Camp  Beale  demonstration  of  portable 
sensors. 


Table  1-1.  Participants  in  the  Portable  Sensor  Demonstration  at  the  former  Camp  Beale 


Task 

Performer(s) 

Task 

Performer(s) 

Site  Preparation 

Parsons 

BUD  Data  Collection 

LBNL 

EM61-Mk2  Data 
Collection  and  Target 
Selection 

Parsons  (with  input  from 
ESTCP) 

MPV  Data  Collection 

Sky  Research 

Intrusive  Investigation 

Parsons 
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1.2  CLASSIFICATION  CONCEPT 

Classification  is  a  process  used  to  make  a  decision  about  the  likely  origin  of  a  signal.  In  the  case  of 
munitions  response,  high-quality  geophysical  data  can  be  interpreted  with  physics-based  models  to 
estimate  parameters  that  are  related  to  the  physical  attributes  of  the  object  that  resulted  in  the  signal, 
such  as  its  physical  size  and  aspect  ratio.  The  values  of  these  parameters  may  then  be  used  to 
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estimate  the  likelihood  that  the  signal  arose  from  an  item  of  interest,  that  is,  a  munition. 
Electromagnetic  Induction  data  are  typically  fit  to  a  three-axis  polarizability  model  that  can  yield 
parameters  that  relate  to  the  physical  size  of  the  object,  its  aspect  ratio,  the  wall  thickness,  and  the 
material  properties.  The  physics  governing  the  electromagnetic  response  of  a  metal  object  is  well 
understood  and  predictable.  Data  collected  with  these  sensors  contain  the  same  information 
content  on  any  site  and  demonstrations  to  date  have  confirmed  that  classification  works  predictably. 

Munitions  are  typically  long,  narrow  cylindrical  shapes  that  are  made  of  heavy-walled  steel. 

Common  clutter  objects  can  derive  from  military  uses  and  include  exploded  parts  of  targets,  such  as 
vehicles,  as  well  as  munitions  fragments,  fins,  base  plates,  nose  cones  and  other  munitions  parts. 
Other  common  clutter  objects  are  man-made  nonmilitary  items.  While  the  types  of  objects  that  can 
possibly  be  encountered  are  nearly  limitless,  common  items  include  barbed  wire,  horseshoes,  nails, 
hand  tools,  and  rebar.  These  objects  and  geology  give  rise  to  signals  that  will  differ  from  munitions 
in  the  parameter  values  that  are  estimated  from  geophysical  sensor  data. 

Once  the  parameters  are  estimated,  a  methodology  must  be  found  to  sort  the  signals  to  identify 
items  of  interest,  in  this  case  munitions,  from  the  clutter.  This  is  termed  classification.  In  a  simple 
situation,  one  can  imagine  sorting  items  based  on  a  single  parameter,  such  as  object  size.  A  rule 
could  be  made  that  all  objects  with  an  estimated  size  larger  than  some  value  will  be  treated  as 
potentially  munitions  items  of  interest,  such  as  large  bombs,  and  those  smaller  could  not  possibly 
correspond  to  intact  munitions. 

In  reality,  many  classification  problems  cannot  be  handled  successfully  based  on  a  single  parameter. 
Because  the  parameter-estimation  process  is  imperfect  and  the  physical  sizes  of  the  objects  of 
interest  may  overlap  with  the  sizes  of  the  clutter  objects,  it  is  rare  to  get  perfect  separation  based  on 
one  parameter.  For  complex  problems,  sophisticated  statistical  classifiers  can  combine  the 
information  from  multiple  parameters  to  make  a  quantitative  estimate  of  the  relative  likelihood  that 
a  signal  corresponds  to  an  item  of  interest. 

1.3  RESULTS  FROM  THE  METALMAPPER  DEMONSTRATION  AT 
CAMP  BEALE 

The  MetalMapper  was  demonstrated  at  Camp  Beale,  concurrent  with  the  demonstration  of  the 
portable  sensors.  This  sensor  is  commercially  available  and  its  use  the  most  widespread  of  the 
advanced  sensors.  It  provides  a  point  of  comparison  for  the  performance  of  the  smaller,  portable 
sensors.  This  demonstration  took  place  in  an  open  and  accessible  area  adjacent  to  the  treed  site 
where  the  portable  sensors  were  demonstrated.  The  targets  of  interest  and  clutter  environment  were 
comparable.  Both  production  contractor  geophysicists  and  the  developers  of  classification  methods 
were  successful  in  using  MetalMapper  data  to  achieve  substantial  classification. 

An  example  of  the  analysis  of  the  MetalMapper  data  is  shown  in  Figure  1-1.  In  this  demonstration, 
there  were  1310  total  clutter  items  as  determined  from  the  ground  truth.  This  analyst  successfully 
identified  all  but  one  of  the  TOI  and  eliminated  about  75%  of  the  clutter.  The  cued  location  from 
the  EM-61  data  for  the  missed  seed  was  not  directly  over  the  item  and  one  of  the  cued  surveys 
moved  further  away  to  another  item  as  part  of  their  optimization;  thus,  the  item  over  which  data  was 
collected  was  not  the  same  as  the  item  dug  leading  to  an  incorrect  classification.  This  classification 
performance  was  typical  for  the  developers  analyzing  the  MetalMapper  data  and  similar  results  were 
achieved  by  some  but  not  all  of  the  production  contractor  analysts.  (Ref.  1) 
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Results  of  SAIC  analysis  of  MetalMapper  data  acquired  by  Parsons  at  the  former  Camp  Beale 


1.4  ABOUT  THIS  REPORT 

This  report  provides  an  overview  of  the  key  results  from  the  demonstration  of  the  portable  systems 
for  project  managers,  regulators,  and  contractors.  All  three  of  these  systems  are  still  in  development 
but  the  results  presented  here  are  indicative  of  their  potential.  The  material  covered  in  this  report 
represents  only  a  small  part  of  a  much  larger  study;  more  information  about  the  entire 
demonstration  can  be  found  in  the  individual  demonstrator  reports  (Refs.  3-12)  and  an  independent 
performance  assessment  by  the  Institute  for  Defense  Analyses.  (Ref.  13) 

The  report  begins  with  a  description  of  the  site  and  an  overview  of  the  program  approach.  We  then 
describe  the  detection  and  classification  performance  of  the  portable  sensors.  This  is  followed  by  a 
discussion  of  costs  and  a  summary  of  the  program  conclusions. 
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2  FORMER  CAMP  BEALE  DEMONSTRATION  SITE 


The  former  Camp  Beale  site  was  selected  for  demonstration  of  the  portable  systems  because  it  is 
partially  wooded  and  exhibits  moderately  hilly  terrain.  It  contains  a  mixture  of  large  and  small 
munitions,  which  presents  a  difficult  classification  problem  likely  to  be  encountered  on  production 
sites.  The  tree  cover  also  increases  the  difficulty  of  obtaining  accurate  GPS  readings,  posing  a 
challenge  for  navigation  and  geolocation  that  will  affect  reacquisition  for  the  cued  systems. 

2.1  SITE  HISTORY  AND  CHARACTERISTICS 

The  site  description  material  reproduced  here  is  taken  from  the  Site  Inspection  Report.  (Ref.  14) 
More  details  can  be  obtained  in  the  report. 

The  former  Camp  Beale  is  an  approximately  60,000-acre  site  located  in  Yuba  and  Nevada  Counties, 
CA.  Camp  Beale  was  subject  to  complex  historical  usage  over  many  years  and  there  is  an 
overlapping  network  of  historical  ranges  throughout.  An  aerial  photo  of  Camp  Beale  showing  the 
location  of  the  demonstration  area  is  shown  in  Figure  2-1. 

The  former  Camp  Beale  property  area  was  acquired  by  the  U.S.  Government  prior  to  1940  and 
consisted  of  85,654  acres.  During  WWII,  it  was  used  by  armored  divisions,  served  as  an  induction 
center  and  a  prisoner  of  war  encampment,  and  was  the  site  for  the  West  Coast  Chemical  Warfare 
School,  among  other  things.  Later  it  was  used  for  a  variety  of  training  activities  by  the  National 
Guard,  the  Air  Force  and  the  Navy.  In  1957  approximately  65,000  acres  was  declared  excess. 

The  demonstration  was  conducted  in  an  area  that  is  located  within  the  historical  bombing  Target  4 
and  the  Proposed  Toss  Bomb  target  area.  An  aerial  photo  of  the  demonstration  area  is  shown  in 
Figure  2-2.  The  area  includes  sections  with  trees  and  steeper  slopes  where  the  man-portable  systems 
surveyed,  in  addition  to  sections  that  are  relatively  flat  and  open,  where  the  MetalMapper 
demonstrations  took  place.  Both  MetalMapper  and  the  portable  systems  were  demonstrated  on  the 
area  marked  Combined.  During  the  demonstration,  it  was  discovered  that  sensor  data  showed 
moderate  geologic  interference.  An  example  of  the  measured  background  variation  in  the  Combined 
Grids  is  shown  in  Figure  2-3.  Variations  of  this  magnitude  make  it  important  to  use  backgrounds 
measured  in  close  proximity  when  analyzing  anomaly  data. 

The  suspected  munitions  in  this  demonstration  area  include,  but  are  not  limited  to: 

•  37-mm  projectiles 

•  60-mm  mortars 

•  81 -mm  mortars 

•  105-mm  projectiles 

At  the  particular  site  of  this  demonstration,  evidence  of  81-mm  mortars  and  105-mm  projectiles  was 
found  during  the  Site  Inspection  intrusive  investigation  in  2005.  It  is  also  suspected  that  60-mm 
mortars  may  be  present.  In  addition,  37-mm  projectiles  have  been  found  scattered  throughout  the 
former  Camp  Beale  and  are  included  as  another  suspected  munition  in  this  area.  Due  to  the  complex 
historical  usage  of  this  site  over  many  years  and  the  overlapping  network  of  historical  ranges 
throughout,  it  is  also  likely  that  other  munitions  types  beyond  those  listed  above  may  be 
encountered. 
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Figure  2-1 .  Aerial  photograph  of  the  former  Camp  Beale  FUDS  with  historic  ranges  overlain.  The  50-acre  site  boundary  which  contains  the 

demonstration  areas  is  shown  in  the  blow-up. 
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Figure  2-2.  The  fifty-acre  demonstration  site  with  the  MetalMapper  grids,  portable  system  grids,  and 

combined  grids  delineated 


Figure  2-3.  Measured  geologic  background  variability  in  the  Combined  Grids  of  the  former  Camp  Beale 

site 
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3  PROGRAM  DESIGN 


3.1  OVERALL  APPROACH 

An  overview  of  the  steps  involved  in  this  demonstration  is  shown  in  the  flow  chart  in  Figure  3-1 . 


Figure  3-1 .  Flow  chart  outlining  steps  in  the  demonstration  at  Camp  Beale.  Blue  boxes  are  tasks 
performed  by  ESTCP.  Others  are  tasks  performed  by  contractors. 

The  objective  of  the  study  was  to  evaluate  classification,  as  opposed  to  detection.  Multiple 
classification  approaches  were  applied  to  data  collected  using  three  different  portable  sensor 
platforms.  For  comparisons  of  different  classification  approaches  to  be  straightforward,  a  common 
set  of  detections  for  each  data  set  was  required.  EM61-MK2  survey  data  were  used  to  perform 
detection.  The  approach  to  detection  is  described  below  in  Section  5.1.  A  common  anomaly  list 
was  passed  to  each  of  the  cued  data  collection  teams  and  all  of  the  analysis  demonstrators. 

All  the  targets  on  the  anomaly  list  were  dug  and  assigned  ground-tmth  labels  designating  whether  or 
not  each  was  a  target  of  interest  (TOI).  These  labeled  data,  including  the  seeded  targets,  were 
available  to  be  used  as  training  data  or  test  data.  Demonstrators  could  choose  to  perform  their 


classification  based  on  no  site  specific  training  data  or  a  demonstrator-requested  training  data  set.  If 
requested,  all  truth  information  for  the  training  data  was  provided  to  the  processors  and  used  to 
train  their  algorithms.  The  truth  labels  for  the  remaining  data  were  sequestered,  and  these  were  used 
for  blind  testing.  The  processors  were  required  to  provide  their  assessment  of  the  TOI/not-TOI 
labels  for  each  item  in  the  test  data  part  of  the  detection  list.  The  labels  were  compared  to  truth  by 
an  independent  third  party  to  score  performance. 

3.2  DEMONSTRATION  PREPARATION 

Several  activities  occurred  prior  to  data  collection  to  ensure  the  resulting  data  would  support  a 
successful  demonstration. 

3.2.1  Seeding  the  Site 

At  a  live  site,  the  number  of  UXO  is  small,  far  from  enough  to  determine  any  demonstrator’s 
classification  performance  with  acceptable  statistical  confidence  bounds.  In  fact,  on  the  Camp  Beale 
demonstration  site,  only  four  munitions  were  recovered  in  the  intrusive  investigation.  Therefore, 
the  site  was  seeded  with  enough  TOI  to  ensure  statistical  validity  on  measures  of  classification  of 
TOI.  The  seeds  are  listed  in  Table  3-1.  For  the  first  time,  the  seeds  included  not  only  inert 
munitions,  but  also  industry  standard  objects.  (Ref.  15)  The  ISOs  are  also  considered  TOI  and 
expected  to  be  both  detected  and  correctly  classified. 


Table  3-1.  Seeds  Emplaced  for  the  Camp  Beale  demonstration  (includes  the 
MetalMapper  demonstration  area) 


Item 

Number 

Depth  range  (cm)* 

Industry  Standard  Object 

Small,  Schedule  40 

65 

10-25 

37-mm  projectile 

59 

10-32 

60-mm  mortar 

34 

15-40 

81 -mm  mortar 

33 

15-50 

105 -mm  projectile 

9 

15-60 

^Depths  are  to  the  center  of  the  object  below  ground  level. 


No  attempt  was  made  to  separate  the  seeds  from  the  surrounding  clutter.  For  safety,  seeds  were 
emplaced  using  standard  anomaly  avoidance  procedures.  For  realism,  the  emplacement  teams  were 
instructed  to  replace  any  metal  dug  up  during  emplacement  back  in  the  hole  with  the  seeded  object. 
All  seed  depths  were  specified  at  1 5  cm  in  the  site  seeding  plan,  with  direction  for  crews  to  bury 
them  deeper  where  soil  conditions  allowed.  In  some  cases,  the  items  were  buried  at  shallower 
depths  due  to  ground  conditions. 

3.2.2  Instrument  Verification  Strip  and  Training 

A  quiet  area  near  the  portable  grids  was  located  to  establish  an  instmment  verification  strip  (IVS)  to 
be  used  for  daily  verification  of  proper  sensor  operation  and  a  training  pit  to  be  used  to  collect 
sensor  data  for  algorithm  training.  Details  of  the  contents  of  the  IVS  are  given  in  Table  3-2. 
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Table  3-2.  Details  of  the  Instrument  Verification  Strip 


Item  ID 

Description 

Depth  (m) 

Inclination 

Azimuth 
(°  cw  from  N) 

T-001 

shotput 

0.30 

N/A 

N/A 

T-002 

105-HEAT 

0.45 

Horizontal 

Across  Track 

T-003 

37-mm  projectile 

0.15 

Horizontal 

Across  Track 

T-004 

60-mm  mortar 

0.15 

Horizontal 

Across  Track 

T-005 

small  ISO 

0.15 

Horizontal 

Across  Track 

3.2.3  Surface  Clearance 

Prior  to  the  digital  mapping  (DGM)  surveys,  Parsons  UXO  personnel  conducted  instrument- aided 
surface  clearance.  The  main  objective  of  the  surface  clearance  was  to  ensure  that  no  hazardous 
items  would  be  encountered  during  the  nonintrusive  phases  in  the  demonstration  area  and  to 
remove  metallic  surface  debris  from  the  grids.  In  addition  to  the  surface  clearance,  Parsons  also 
conducted  minor  brush  clearing,  cutting  low  branches  and  removed  fallen  trees  from  the 
demonstration  area.  This  operation  required  one  week.  The  majority  of  items  found  on  the  surface 
sweeps  were  barbed  wire,  along  with  small  munitions  fragments.  Two  notable  items  identified 
during  the  surface  clearance  were  an  empty  75-mm  projectile  and  a  large  pile  of  barbed  wire  that  was 
eventually  moved  out  of  the  survey  area.  These  are  shown  in  Figure  3-2. 


Figure  3-2.  Examples  of  the  items  removed  during  the  pre-survey 
surface  clearance. 


3.3  TARGETS  OF  INTEREST 

The  main  goal  of  classification  in  the  pilot  program  is  to  identify  with  high  confidence  items  that  can 
be  safely  left  behind.  At  Camp  Beale,  the  project  team  determined  that  targets  of  interest  that  must 
be  removed  would  include: 

•  seeded  munitions  and  Industry  Standard  Objects  and 

•  intact  munitions  recovered  at  the  site,  both  live  and  inert. 
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Eighty  eight  items  were  seeded  in  the  areas  surveyed  with  the  portable  systems  and  all  are  TOI. 

Two  37-mm  projectiles  were  recovered  in  these  areas  that  were  classified  by  the  UXO  specialists  as 
munitions  debris  because  they  were  empty.  Both  of  these  projectiles  were  intact  (Figure  3-3)  so  they 
were  deemed  TOI  for  this  study. 


Figure  3-3.  Two  "empty"  37-mm  projectiles  recovered  in  this  demonstration. 


3.4  DATA  COLLECTION 


The  classification  pilot  study  tested  combinations  of  data-collection  platforms  and  analysis 
approaches.  Data-collection  plans  were  generated  by  all  data  collectors  and  shared  with  the  data 
processors  prior  to  deployment. 

An  EM61  instmment  was  used  to  collect  high  quality,  high  density  survey  data.  These  data  were 
used  both  for  anomaly  detection  and  to  test  what  can  be  achieved  attempting  classification  with 
careful  data  collection  using  standard  equipment  and  field  techniques.  Data  were  collected  with  a 
standard  cart  platform  EM61-MK2  system  in  the  four-channel  mode.  The  sensor  height  above 
ground  was  the  standard  40  cm  to  the  bottom  of  the  coil  housing.  Positioning  was  accomplished 
using  either  RTK  GPS,  Trimble  Robotic  Total  Station  (RTS),  or  fiducial  methods,  depending  on  the 
availability  GPS  signal  and  tree  density  in  the  wooded  area.  (Ref.  6)  Data  were  acquired  by  mnning 
the  sensor  in  closely  spaced  lines,  similar  to  the  pattern  of  a  lawnmower  cutting  grass.  The  site  was 
divided  into  30-m  x  30-m  grids,  Figure  2-2,  and  data  collected  one  grid  at  a  time  with  survey  lanes 
spaced  at  0.5-m  intervals. 

The  three  prototype  portable  systems  specially  designed  to  maximize  classification  of  munitions 
collected  data  over  each  anomaly  detected  by  the  EM61.  The  geophysical  sensors  are  described 
briefly  below  in  Section  4.  Details  may  be  found  in  the  reports  provided  by  the  performers  (Refs. 
7-9).  Each  anomaly  in  the  detection  survey  was  reacquired  using  either  GPS  or  RTS.  To  account 
for  the  relatively  poorer  geolocation  in  the  survey  data  taken  in  the  woods,  the  positions  were 
refined  by  finding  the  peak  of  the  EM61  signature.  This  location  was  marked  with  a  flag  that  that 
was  used  to  position  the  cued  data  collection  for  all  three  advanced  sensors. 

3.5  CLASSIFICATION  APPROACHES 
3.5.1  Geophysical  Models 

Classification  demonstrators  could  analyze  the  survey  data,  the  cued  data,  or  a  mix  of  the  two;  for 
some  anomalies  in  some  analysis  schemes  decisions  could  be  made  from  the  survey  data  so  there 
was  no  need  to  bear  the  cost  of  a  cued  measurement.  The  data  corresponding  to  each  anomaly  were 
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analyzed  by  the  processing  teams  to  extract  parameters  by  fitting  the  data  to  a  geophysical  model. 
This  has  the  effect  of  separating  the  intrinsic  target  parameters  from  extrinsic  variables  such  as  the 
distance  and  orientation  between  the  sensor  and  the  target.  All  but  one  of  the  processing 
approaches  relied  on  the  dipole  model. 

Some  of  the  intrinsic  parameters  that  were  considered  included: 

•  the  electromagnetic  polarizabilities,  which  relate  to  the  object’s  physical  size  and  aspect  ratio, 
and 

•  the  electromagnetic  decay  constants,  which  relate  to  the  object’s  material  properties  and  wall 
thickness. 

3.5.2  Classifiers 

Once  the  parameters  are  estimated,  a  mechanism  is  needed  to  decide  whether  the  corresponding 
object  is  a  target  of  interest  or  not.  Several  types  of  classification  processing  schemes  were  evaluated 
in  the  classification  study.  These  included  both 

Statistical  classification:  Computer  algorithms  evaluate  the  contributions  of  each  parameter  to 
defining  munitions  likeness  based  on  “training”  on  a  subset  of  the  data  for  which  the  identities 
of  the  objects  are  known.  Then  the  unknown  objects  are  prioritized  based  on  whether  their 
parameters  are  statistically  similar  to  known  objects  in  the  training  data. 

Library-matching  classification:  One  or  more  of  the  three  polarizability  decay  curves  derived 
for  each  anomaly  are  compared  to  a  library  of  responses  of  munitions  and  surrogates.  The 
unknowns  are  prioritized  based  on  how  well  they  match  one  of  the  library  responses. 

Most  classification  algorithms  require  some  training  to  select  the  parameters  or  features  that  are 
most  useful  for  classification  and  set  thresholds  in  the  decision  process.  For  this  demonstration,  the 
analysts  had  the  choice  of  using  training  data  previously  collected  at  other  sites  only,  supplementing 
those  data  with  data  from  the  IVS  and  training  pit,  or  adding  training  data  obtained  from  excavation 
of  a  limited  number  of  anomalies  from  the  site.  For  those  demonstrators  that  chose  to  use  on-site 
training  data,  the  anomaly  list  was  divided  into  training  and  blind  testing  sets;  for  those  who  did  not 
choose  on-site  training,  the  test  set  consisted  of  all  anomalies.  After  training,  the  decision  process 
for  each  algorithm  was  finalized  and  documented,  and  the  demonstrators  provided  ranked  dig  lists 
for  the  blind  test  set. 

The  final  step  in  classification  is  delineating  the  targets  of  interest  from  those  that  are  not.  For 
example,  in  the  case  of  a  statistical  classifier,  all  the  anomalies  are  ordered  by  the  likelihood  that  they 
belong  to  the  class  of  the  targets  of  interest.  These  likelihood  values  do  not  represent  a  yes/no 
answer,  but  rather  a  continuum  within  which  a  dividing  line  or  threshold  must  be  specified. 
Depending  on  the  application,  this  threshold  may  be  set  to  try  to  avoid  false  positives,  which  may 
come  at  the  expense  of  missing  some  items  of  interest,  or  it  may  be  set  to  try  to  avoid  false 
negatives,  which  will  come  at  the  expense  of  a  greater  number  of  non-TOI.  In  this  program,  where 
missing  an  item  of  interest  represented  the  most  serious  failure,  demonstrators  selected  thresholds  to 
try  to  retain  all  the  detected  munitions. 


12 


3.6  CLASSIFICATION  PRODUCT 


Demonstrators  were  asked  to  produce  a  ranked  anomaly  list  for  each  sensor  and  processing 
combination.  These  lists  were  constructed  as  shown  in  Figure  3-4. 


Initial  Ranked  Anomaly  List 


Rank 

Anomaly 

ID 

Decision 

Statisitic 

Dig 

Comment 

1 

2498 

-9999 

Y 

Can't  extract  reliable  parameters 

5 

811 

Y 

First 

313 

N 

Unable  to  classify 

Threshold 

883 

N 

N 

N 

N 

High  likelihood  not  TOI 

N 

N 

N 

.02 

N 

.02 

N 

N 

.01 

N 

Final  Ranked  Anomaly  List 


Figure  3-4.  Format  for  the  initial  (left)  and  final  (right)  ranked  anomaly  lists  for  this  demonstration 


GRAY:  Targets  where  the  signal-to-noise  ratio  (SNR),  data  quality,  or  other  factors 
prevent  any  meaningful  analysis  were  deemed  “can’t  extract  reliable  parameters”  and 
placed  at  the  top  of  the  list. 


The  next  items  were  those  that  the  demonstrator  was  most  certain  are  TOI. 


YELLOW:  A  band  was  specified  indicating  the  targets  where  the  data  can  be  fit  in  a 
meaningful  way,  but  the  derived  parameters  do  not  permit  a  high  confidence 
determination  of  TOI  or  not-TOI  Some  of  these  anomalies  may  be  marked  to  be  dug 
in  the  first  round  to  resolve  these  ambiguities. 


GREEN:  The  bottom  item  in  the  list  was  that  which  the  demonstrator  was  most 
certain  does  NOT  correspond  to  a  TOI. 


•  THRESHOLD:  A  threshold  was  set  at  the  point  beyond  which  the  demonstrator 

would  recommend  all  anomalies  be  treated  as  TOI,  either  because  they  are  determined 
to  be  so  with  high  confidence  or  because  a  high-confidence  determination  that  they  are 
not  TOI  cannot  be  made.  This  is  indicated  by  the  heavy  black  dashed  line. 


Demonstrators  were  allowed  to  submit  several  successive  dig  lists  to  refine  their  analysis  but  the 
final  list  had  to  include  a  dig/ no-dig  judgment  for  every  anomaly. 
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3.7  SCORING  METHODS 

The  demonstration  was  scored  based  on  the  demonstrator’s  ability  to  eliminate  nonhazardous  items 
while  retaining  all  detected  TOI.  A  common  way  to  evaluate  performance  of  detection  and 
classification  is  the  receiver  operating  characteristic  (ROC)  curve.  An  example  is  shown  in  Figure 
3-5.  The  colored  regions  on  the  plot  in  Figure  3-5  correspond  to  the  colors  of  the  various  sections 
of  the  ranked  dig  list  in  Figure  3-4.  The  ROC  curve  is  a  plot  of  the  percent  of  the  TOI  dug,  that  is  it 
reflects  the  probability  of  correctly  classifying  the  detected  munitions  items,  versus  the  number  of 
non-TOI.  A  perfect  classifier  would  correctly  identify  100%  of  the  munitions  and  no  clutter.  We 
have  modified  the  traditional  ROC  curve  slightly  to  reflect  both  the  TOI  and  non-TOI  dug  for 
training.  This  is  done  to  account  for  the  fact  that  different  methods  used  different  amounts  of 
training  data. 
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Figure  3-5.  Example  receiver  operating  characteristic  curve. 

The  key  regions  to  interpret  the  ROC  curves  used  in  this  program  are: 

•  A:  Targets  to  the  left  and  below  this  point  were  dug  for  training  data.  Site  specific  training 
data  were  used  in  many  of  the  processing  approaches  and  these  digs  would  be  required. 
Different  approaches  required  differing  amounts  of  training  data;  the  ROC  curves  for  those 
that  used  no  site-specific  training  data  start  at  the  origin. 

•  B:  Targets  from  point  A  to  this  point  were  categorized  as  can’t  analyze  and  would  need  to  be 
treated  as  potential  TOI  because  no  meaningful  classification  could  be  done.  In  this 
example,  about  30  of  the  can’t  analyze  targets  were  false  positives,  reflected  in  the  position 
of  the  point  on  the  horizontal  axis.  No  TOI  were  included  in  the  can’t  analyze  list. 

•  C:  In  the  absence  of  any  classification,  this  sensor  detected  all  the  TOI  and  had  more  than 
2100  non-TOI  items  in  the  detection  list. 

•  D:  Based  on  classification,  this  is  the  demonstrator’s  threshold  for  the  dividing  point 
between  TOI  and  not-TOI.  This  demonstrator  missed  one  TOI  at  her  threshold. 

•  E:  This  demonstrator’s  best  threshold  chosen  retrospectively.  If  the  threshold  had  been 
chosen  perfectly,  only  200  targets  could  have  been  left  in  the  ground. 
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4  ADVANCED  SENSOR  TECHNOLOGIES 


Three  portable  sensors  were  used  to  collect  cued  data  at  the  locations  of  anomalies  detected  by  the 
EM61  cart.  These  purpose-built  EMI  systems  differ  from  the  EM61  is  two  important  ways.  First, 
they  were  designed  to  collect  sufficient  data  to  fully  characterize  the  EMI  signature  of  a  buried 
object  from  a  single  measurement  location  or  a  series  of  closely-spaced  static  measurements. 

Second,  they  collect  a  more  complete  measurement  of  the  time  response  of  the  target,  both  farther 
out  in  time  and  at  greater  fidelity. 

4.1  MAN-PORTABLE  VECTOR  SENSOR  (MPV) 

The  MPV  is  a  time-domain,  EMI  sensor  composed  of  a  single  transmitter  coil  and  an  array  of  five 
receiver  units  that  measure  all  three  components  of  the  EM  field  as  shown  in  Figure  4-1.  The  MPV 
sensor  head  for  this  demonstration  comprised  a  50-centimenter  (cm)  diameter  circular  loop 
transmitter  coiled  around  a  disk  that  intermittently  illuminates  the  subsurface,  and  five  8-cm  multi- 
component  receiver  units  (cubes)  that  measure  the  three  orthogonal  components  of  the  transient 
secondary  EM  field  decay. 


attitude  sensor 


Touch-screen 
control  display 


Beacon  boom  for  positioning  system 


MPV  sensed  head 


Figure  4-1 .  Components  of  the  MPV.  Left  inset  shows  data  acquisition  (DAQ)  and  power  unit  mounted  on 
a  backpack  frame.  Right  panel  shows  view  of  sensor  head  from  above  with  cube  numbers. 

Cued  data  were  collected  in  a  9-point  grid  around  the  flagged  anomaly  location  using  the  MPV- 
beacon  positioning  system  to  obtain  local  sensor  positions.  (Ref.  7)  The  positioning  system  works 
by  locating  the  origin  of  the  primary  field  generated  by  the  MPV  transmitter  coil,  acting  as  a  beacon, 
with  a  pair  of  EMI  receivers  rigidly  attached  to  a  portable  beam,  placed  horizontally  on  the  ground 
and  supported  by  a  pair  of  tripods  to  act  as  a  base  station.  The  azimuth  of  the  MPV  and  boom  are 
recorded  with  3-component  attitude  sensor.  Decay  data  were  collected  to  25  ms  after  primary  field 
turn-off  for  this  survey.  A  photo  of  MPV  data  collection  at  Beale  is  shown  in  Figure  4-2. 
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Figure  4-2.  MPV  data  collection  in  the  trees  at  Camp  Beale 
4.2  TEMTADS  2X2 

The  TEMTADS  2x2  array  is  comprised  of  four  individual  EMI  transmitters  with  3-axis  receivers, 
arranged  in  a  2  x  2  array  as  shown  in  Figure  4-3.  The  center- to-center  distance  is  40  cm,  yielding  an 
80  cm  x  80  cm  array.  The  data  acquisition  computer  is  mounted  on  a  backpack  worn  by  one  of  the 
data  acquisition  operators.  The  second  operator  controls  the  data  collection  using  a  personal  data 
assistant  (PDA)  which  wirelessly  communicates  with  the  data  acquisition  computer.  The  second 
operator  also  manages  field  notes  and  team  orienteering  functions. 


EM  Sensor 


Figure  4-3.  Schematic  of  the  TEMTADS  2x2  array 

For  each  series  of  measurements  with  the  array,  the  four  transmitters  are  energized  sequentially. 
After  each  excitation  pulse,  the  response  of  all  twelve  receive  coils  is  recorded,  resulting  in  48 
(4x4x3)  transmit/ receive  pairs.  Data  were  recorded  for  25  ms  after  transmitter  turn-off.  (Ref.  8) 
A  photograph  of  the  TEMTADS  2x2  array  collecting  data  at  Camp  Beale  is  shown  in  Figure  4-4. 
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Figure  4-4.  Portable  TEMTADS  2x2  data  acquisition  at  Beale 
4.3  BERKELEY  UXO  DISCRIMINATOR  (BUD) 

The  prototype  hand-held  UXO  discriminator  employs  three  orthogonal  transmitters  and  ten  pairs  of 
differenced  receivers  in  a  35-cm  cube.  Figure  4-5.  Each  vertical  face  of  the  cube  has  three  induction 
coils,  and  two  horizontal  faces  have  four  induction  coils,  each  sensitive  to  the  magnetic  field 
component  normal  to  the  face.  Receivers  on  opposite  faces  of  the  cube  are  paired  along  the 
symmetry  lines  through  the  center  of  the  system  and  each  pair  sees  identical  fields  during  the  on- 
time  of  current  pulses  in  the  transmitter  coils.  The  pairs  are  wired  in  opposition  to  produce  zero 
output  during  the  on— time  of  the  pulses  in  three  orthogonal  transmitters.  This  configuration 
dramatically  reduces  noise  in  measurements  by  canceling  background  electromagnetic  fields  (these 
fields  are  uniform  over  the  scale  of  the  receiver  array  and  are  consequently  nulled  by  the  differencing 
operation),  and  by  canceling  noise  contributed  by  the  tilt  of  the  receivers  in  the  Earth’s  magnetic 
field.  Thus,  the  gradiometer  configuration  greatly  enhances  receivers’  sensitivity  to  the  target 
response. 


Figure  4-5.  Transmit  and  receive  coil  configuration  for  the  prototype  portable  UXO  discriminator 

At  Beale,  cued  data  were  collected  at  each  flag,  and  at  0.15  m  before  and  0.15  m  after  it,  with  the 
system  oriented  in  a  single  direction.  Soundings  were  differenced  with  background  reference 
soundings  taken  within  the  previous  30-40  minutes  at  a  nearby  site  determined  by  the  field  operator 
to  be  free  from  metallic  objects.  When  polarizability  inversions  from  all  three  soundings  were 
suspected  to  arise  from  more  than  one  object,  an  additional  two  soundings  were  taken,  one  on  each 
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side  of  the  system,  0.15m  from  the  flag.  (Ref.  9)  Decay  data  were  collected  at  46  sample  times 
logarithmically  spaced  from  80  to  1,460  ps.  A  photograph  of  the  system  in  use  at  Camp  Beale  is 
shown  in  Figure  4-6. 


Figure  4-6.  Portable  Berkeley  UXO  Discriminator  at  Camp  Beale 
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5  ANOMALY  SELECTION  AND  INVESTIGATION  RESULTS 


5.1  ANOMALY  SELECTION 

After  the  survey  sensor  completed  data  acquisition,  anomalies  were  selected  from  the  data  using  a 
procedure  designed  by  the  program  office.  Since  this  sensor  detection  list  was  the  basis  for  all 
subsequent  analyses,  a  rigorous  process  was  used  to  set  this  threshold. 

5.1.1  Anomaly  Selection  Threshold 

The  known  targets  of  interest  in  this  demonstration  were  105-mm  and  37-mm  projectiles  and  60- 
mm  and  81 -mm  mortars.  Of  these,  the  37  mm  is  the  most  difficult  to  detect.  Prior  to  the 
demonstration,  the  site  team  determined  that  detection  of  37-mm  projectiles  to  1  foot  (30  cm)  depth 
was  the  objective  for  this  demonstration.  Accordingly,  the  anomaly  selection  threshold  was  set  as 
the  smallest  signal  expected  from  a  37-mm  projectile  at  30  cm  depth. 

Figure  5-1  illustrates  this  process.  The  predicted  signal  from  the  EM61-MK2  for  37-mm  projectiles 
(Ref.  16)  in  their  least  and  most  favorable  orientations  is  plotted  in  the  figure  along  with  a  vertical 
line  marking  the  30  cm  depth  of  interest.  The  gate  2  anomaly  selection  threshold  for  this  sensor 
system  was  set  at  5.2  mV  based  on  this  curve.  Also  plotted  on  Figure  5-1  is  the  observed  noise  in 
the  cued  area.  As  can  be  seen  from  the  figure,  the  anomaly  selection  threshold  is  well  above  the 
measured  noise  so  the  anomaly  selection  process  should  be  relatively  unambiguous  for  this  sensor 
system.  This  threshold  will  detect  the  other  TOI  to  the  depths  in  Table  5-1. 


Depth  (cm  bgs)  as  Deployed  on  Standard  Wheels 
0  20  40  60  80 


Figure  5-1.  Predicted  EM61-MK2  anomaly  amplitude  in  gate  2  for  a  37-mm  projectile  in  its  least  and  most 
favorable  orientations.  Also  shown  are  the  RMS  noise  measured  at  the  site,  the  30  cm  depth  used  to  set 
the  threshold  and  the  anomaly  selection  threshold  used  in  this  demonstration. 
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Table  5-1.  Depths  to  which  TOI  on  Camp  Beale 
will  be  detected  at  a  threshold  of  5.2  mV 


Munitions 

Depth/ cm 

Small  ISO,  Schedule  40 

24 

37-mm  projectile 

30 

60-mm  mortar 

60 

81 -mm  mortar 

65 

105 -mm  projectile 

110 

Many  targets,  especially  those  that  with  high  length  to  diameter  aspect  ratios,  result  in  multiple, 
closely-spaced  exceedances.  To  avoid  having  redundant  locations  on  the  final  anomaly  list,  all 
exceedances  within  the  distance  of  0.6  m  were  grouped  into  a  single  detection.  Finally,  all  pairs  of 
exceedances  between  0.6  m  and  1.0  m  apart  were  examined  by  a  trained  analyst  who  made  a 
judgment  whether  they  corresponded  to  a  single  source  or  not.  This  consolidation  resulted  in  91 1 
identified  anomalies  in  the  grids  surveyed  by  the  portable  instruments  with  all  seeds  detected. 

5.2  GEOLOCATION  ACCURACY 

Because  of  the  difficulties  with  precise  navigation  and  geolocation  in  the  wooded  area,  the  anomaly 
locations  were  offset  from  the  true  seed  locations  by  larger  distances  than  are  typical  for  surveys  in 
open  areas.  A  histogram  of  these  offsets  is  shown  in  the  left  panel  of  Figure  5-2.  All  of  the  seeds 
were  located  within  1  m  of  the  EM61  anomaly,  but  many  are  at  distances  in  excess  of  the  0.4  m 
specification  typically  used  to  acquire  cued  data  with  the  advanced  sensors.  For  comparison,  the 
results  achieved  in  the  open  at  Camp  Butner  are  shown  in  the  right  panel  of  Figure  5-2.  (Ref  17)  To 
ensure  that  the  cued  data  was  taken  over  the  actual  targets  locations,  all  anomalies  were  relocated  by 
first  acquiring  the  location  of  the  anomaly  using  GPS  or  RTS.  These  locations  were  then  refined 
using  an  EM61  with  the  ultimate  flag  location  determined  by  the  peak  of  the  EM61  signal. 
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Figure  5-2.  Comparison  of  the  distance  between  seed  location  and  nearest  anomaly  location  in  the  EM61 
survey  data  from  the  treed  area  at  Camp  Beale  (left)  and  Camp  Butner  (right) 
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5.3  INTRUSIVE  INVESTIGATION 


The  distribution  of  recovered  items  by  class  is  detailed  in  Table  5-2.  Except  for  the  TOI,  all  these 
items  are  classified  as  clutter.  As  at  most  sites,  the  vast  majority  of  recovered  items  were  classified  as 
munitions  debris.  The  fuze  parts  were  expended  and  thus  classified  by  the  UXO  technicians  as 
munitions  debris;  they  are  listed  separately  for  information  only.  Representative  examples  of  the 
clutter  recovered  are  pictured  in  Figure  5-3. 

Table  5-2.  Distribution  of  recovered  items  from  this  demonstration 


Classification 

Number  of  Anomalies 

Targets  of  Interest 

93 

Munitions  Debris 

965 

Fuze  parts 

40 

Cultural  Debris 

21 

Soil/No  Contact 

36 

Total 

1155 

Figure  5-3.  Examples  of  items  recovered  in  this  demonstration.  These  items  were  classified  as  cultural 
debris  (left),  a  fuze  part  (center),  and  munitions  debris  (right). 


The  measured  depths  of  the  recovered  items  (other  than  seeds)  are  plotted  in  Figure  5-4.  As 
expected,  most  recovered  items  were  quite  shallow;  93%  of  all  recoveries  corresponded  to  less  than 
1 0  cm  to  the  center  of  the  target  and  nothing  was  found  deeper  than  20  cm.  These  results  confirm 
our  expectation  that  detection  of  37-mm  projectiles  at  30-cm  (1-foot)  depth  was  a  reasonable  goal 
for  this  site.  The  other  TOI  would  have  been  detected  at  depths  between  60  cm  and  1.1  m,  as 
shown  in  Table  5-1,  although  none  of  these  items  were  recovered  in  this  demonstration. 
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Depth  (cm) 

Figure  5-4.  Histogram  of  recovery  depths  for  all  items  other  than  seeds 
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6  PERFORMANCE  OF  THE  PORTABLE  SENSORS 


This  demonstration  was  the  first  deployment  of  the  portable  sensors  to  a  live  munitions  site.  The 
primary  measure  of  their  performance  is  the  classification  results  discussed  in  the  next  section,  but  it 
is  instructive  to  compare  the  intrinsic  target  parameters  derived  from  the  portable  sensor’s  data  to 
that  of  the  baseline  MetalMapper  sensor  on  the  common  targets  measured  in  the  Combined  Grids 
(Figure  2-2)  and  the  Instrument  Verification  Strip.  Figure  6-1  plots  the  polarizability  decay  curves 
for  four  of  the  IVS  targets  from  each  of  the  four  instruments. 


0.1  1  10 


Figure  6-1 .  Polarizabilities  for  four  items  from  the  IVS  derived  from  measurements  of  the  three  portable 
sensors  used  in  this  demonstration  compared  to  those  derived  using  the  MetalMapper 


Although  the  sensors  collect  decay  data  for  different  times  (1.4  ms  for  BUD,  8  ms  for  MetalMapper, 
and  25  ms  for  MPV  and  TEMTADS  2x2),  the  polarizabilities  for  the  common  times  are  remarkably 
similar  among  the  four  sensors  meaning  that  any  classification  decision  based  on  polarizabilities  will 
be  the  same  for  each  of  the  instruments.  The  late  time  (>  5  or  10  ms)  data  for  the  smaller  objects  is 
noisy  but  that  will  have  minimal  effect  on  the  classification  performance. 


The  three  portable  sensors  and  their  survey  crews  have  had  different  levels  of  field  experience.  In 
addition,  the  MPV  and  portable  BUD  sensors  acquired  cued  data  at  multiple  positions  per  anomaly. 
These  two  factors  result  in  the  daily  productivity  listed  in  Table  6-1 . 


Table  6-1 .  Average  daily  productivity  for  the  portable  sensor  systems 


Sensor 

Average  Anomalies  per  Day 

MPV 

90 

TEMTADS  2x2 

175 

Portable  BUD 

90 
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7  CLASSIFICATION  RESULTS 


A  number  of  the  demonstrators  investigated  multiple  methods  for  training,  parameter  estimation, 
and  classification  during  this  demonstration.  As  a  result,  a  total  of  19  dig  lists  were  scored  in  the 
blind  phase  of  the  demonstration,  representing  the  various  combinations  of  sensor  data  collection 
systems  and  processing  approaches  used.  All  of  the  results  can  be  found  in  the  report  by  IDA  (Ref. 
13).  In  the  following  sections,  we  present  selected  results  that  illustrate  important  conclusions  of  the 
demonstration,  focusing  on  what  can  be  achieved  with  currently  available  technologies  and  the  value 
added  of  the  advanced  portable  sensors  and  processing. 

The  results  in  this  section  are  presented  as  ROC  curves,  which  plot  the  percent  of  correcdy 
classified  munitions  versus  the  number  of  false  positives  (i.e.,  unnecessary  digs).  Their  interpretation 
is  described  in  Section  3.7.  The  colored  segments  of  the  ROC  curve  correspond  to  the  categories 
specified  on  the  dig  list  and  two  threshold  values  are  shown  on  the  ROCs.  The  dark  blue  dot  (•) 
indicates  the  demonstrator’s  threshold  beyond  which  all  targets  are  considered  high  confidence  non- 
TOI.  The  orange  dot  (•)  indicates  the  best  that  the  demonstrator  could  have  done  had  the 
threshold  been  set  in  the  optimal  place,  the  point  at  which  the  first  TOI  would  be  incorrectly 
classified  as  non-TOI,  which  would  produce  the  first  false  negative.  Missed  TOIs  on  the  ROC 
curve  are  indicated  by  open  black  triangles  (A). 

7.1  EM61-MK2  CART 

The  EM61-MK2  is  the  most  common  geophysical  sensor  in  use  for  Munitions  Response  (MR) 
projects  today.  This  sensor  on  a  cart  platform  is  our  benchmark  for  what  could  be  accomplished 
with  carefully  collected  production  geophysics  data.  These  data  were  analyzed  by  two  of  the 
algorithm  developers  participating  in  the  demonstration.  An  SAIC  analyst  used  the  UX- Analyze 
module  of  Oasis  montaj  to  rank  the  anomalies  based  on  electromagnetic  decay  with  total 
polarizability  used  to  resolve  a  few  anomalies.  The  ROC  curve  resulting  from  this  analysis  is  shown 
in  Figure  7-1. 
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Figure  7-1.  SAIC  analysis  of  the  EM61-MK2  cart  data  using  UX-Analyze 
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As  at  other  sites  contaminated  with  37-mm  projectiles  mixed  with  fragments  of  larger  munitions,  the 
EM61  data  were  not  very  useful  for  classification  at  this  site.  The  best  possible  performance  for  this 
sensor  and  analysis  combination,  denoted  by  the  orange  dot  (•),  would  only  correspond  to  a  little 
more  than  200  avoided  clutter  digs. 


The  EM61-MK2  cart  data  were  also  analyzed  by  Sky  Research.  Sky  used  a  combination  of  the  total 
polarizability  and  the  polarization  decay  as  inputs  to  a  statistical  classifier.  These  results  of  this 
analysis  are  shown  in  Figure  7-2.  Although  in  retrospect,  the  best  possible  performance  from  this 
analysis  would  have  avoided  ~450  clutter  digs,  the  analyst  was  not  able  to  confidently  classify  any  of 
the  anomalies  as  “high  likelihood  clutter.” 
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Figure  7-2.  ROC  curve  resulting  from  Sky's  statistical  classifier  applied  to  the  EM61-MK2  cart  data 
7.2  MPV 


Three  analyses  of  the  MPV  cued  data  are  shown  in  Figures  7-3  through  7-5.  The  team  that  collected 
the  data  used  a  an  analysis  that  removed  the  soil  background  contribution  before  inverting  the 
geophysical  data  (Ref.  7)  then  ranked  the  anomalies  using  a  statistical  classifier  with  the  results 
shown  in  Figure  7-3.  At  their  threshold,  they  were  able  to  correctly  classify  about  80%  of  the  clutter 
while  identifying  100%  of  the  TOI.  Retrospective  analysis  shows  that  their  threshold  was 
conservative;  even  better  performance  was  possible. 


Two  other  approaches  to  the  analysis  of  these  data  were  taken  by  SAIC  and  Dartmouth.  SAIC  used 
a  dipole  model  (Ref.  1 1)  to  extract  target  features  while  Dartmouth  used  a  more  advanced  model 
(Ref.  12).  The  results  of  these  two  analyses  are  shown  in  Figures  7-4  and  7-5.  Notice  that  both  of 
these  analysts  requested  substantially  more  training  data  that  the  Sky  analysts;  it  was  these  two 
groups  first  time  working  with  blind  data  from  the  MPV.  After  accounting  for  these  training  data, 
the  classification  was  relatively  efficient;  the  SAIC  ROC  curve  in  particular  is  nearly  vertical  in  the 
red  portion. 
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Figure  7-3.  ROC  curve  resulting  from  the  Sky  analysis  of  MPV  cued  data  collected  at  Camp  Beale 
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Figure  7-4.  ROC  curve  resulting  from  the  SAIC  analysis  of  MPV  cued  data  collected  at  Camp  Beale 
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Figure  7-5.  ROC  curve  resulting  from  the  Dartmouth  analysis  of  MPV  cued  data  collected  at  Camp  Beale 


7.3  TEMTADS  2X2  ARRAY 

Results  of  analyses  of  the  TEMTADS  2x2  array  cued  data  by  Sky  and  SAIC  are  shown  in  Figure  7-6 
and  Figure  7-7  respectively.  The  Sky  team  used  features  resulting  from  geophysical  inversion  of  the 
measured  data  using  a  model  that  handles  multiple  sources  as  input  to  a  support  vector  machine 
classifier.  These  analysts  were  able  to  correctly  label  ~650  targets  as  clutter  with  no  missed 
munitions. 
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Figure  7-6.  Sky  analysis  of  the  TEMTADS  2x2  cued  data  collected  at  Camp  Beale 

Similar  results  were  obtained  from  the  SAIC  analysis  of  the  TEMTADS  data,  Figure  7-7.  This 
analysis  involved  model  fits  to  a  multi-source  model  with  a  classifier  that  was  rule-based.  This 
analysis  resulted  in  ~25  anomalies  classified  as  “can’t  extract  reliable  parameters.”  As  with  the  MPV 
data,  both  of  these  analysts  requested  more  than  50  on-site  training  labels. 
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Figure  7-7.  SAIC  analysis  of  the  TEMTADS  2x2  cued  data  collected  at  Camp  Beale 


7.4  PORTABLE  BUD 

Two  approaches  to  analyze  the  portable  BUD  data  were  used  by  Sky  analysts  and  their  results  are 
shown  in  Figure  7-8  and  Figure  7-9,  respectively.  Figure  7-8  shows  the  results  of  a  library-match 
classifier  and  Figure  7-9  plots  the  results  of  an  analysis  using  a  statistical  classifier  similar  to  the 
TEMTADS  2x2  analysis  presented  in  Figure  7-6. 
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Figure  7-8.  Sky  library-match  analysis  of  the  BUD  cued  data 


The  underlying  feature  extraction  was  similar  for  both  of  these  analyses.  Both  analysts  requested 
about  25  training  labels  and  had  a  significant  number  of  anomalies  they  were  unable  to  analyze. 
Both  analysts  were  also  very  conservative  in  their  thresholds;  retrospective  analysis  shows  the 
potential  for  better  performance  in  both  cases. 
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Figure  7-9.  Sky  analysis  of  BUD  cued  data  using  a  statistical  classifier 


7.5  RESULTS  OVERVIEW 

An  overview  of  all  the  results  from  all  analyses  of  advanced  sensor  data  collected  during  this 
demonstration  is  shown  in  Figure  7-10.  All  sixteen  analysts  were  able  to  correctly  identify  100%  of 
the  TOI.  The  blue  bars  represent  the  number  of  clutter  items  dug  at  each  analyst’s  operating 
threshold.  The  worst  performer  was  able  to  correctly  classify  ~40%  of  the  clutter  correctly.  The 
best  analysts  were  able  to  reject  >  75%  of  the  clutter  using  data  from  any  of  the  three  sensors.  The 
orange  bars  show  the  number  of  clutter  items  that  the  analyst  needed  to  dig  to  just  get  to  1 00% 
correct  classification  of  the  TOI,  regardless  of  where  the  analyst  put  the  threshold.  This  can  be 
thought  of  as  the  best  the  analyst  could  have  done  by  putting  the  threshold  in  exactly  the  optimum 
place  where  the  last  TOI  is  found  in  the  ranked  list. 


Analyst 

Figure  7-1 0.  Overview  of  the  performance  of  all  analyses  of  the  cued  data  from  the  advanced  portable 
sensors  at  Camp  Beale.  All  analysts  correctly  identified  100%  of  the  TOI.  The  blue  bars  represent  the 
number  of  clutter  items  dug  at  the  analysts  chosen  threshold.  The  orange  bars  represent  the  best 
performance  possible  from  each  sensor/analysis  combination. 


29 


8  COST  SAVINGS 


The  demonstration  took  place  on  a  small  part  of  the  former  Camp  Beale  site  and  incurred  costs  for 
many  items  specific  to  a  demonstration  that  would  not  be  needed  in  an  application  of  classification 
to  a  real  site.  Nevertheless,  we  can  extract  meaningful  projected  performance  for  the  technology 
and  apply  reasonable  industry  unit  costs  for  various  elements  to  arrive  at  a  total  cost  comparison  for 
clearing  an  example  100-acre  wooded  site  with  and  without  the  use  of  classification. 

We  made  the  following  assumptions: 

•  The  example  takes  place  in  an  area  with  similar  munitions  types  and  the  same  density  of 
anomalies  as  seen  in  the  demonstration.  Excluding  the  seeds,  there  were  822  anomalies  in 
the  4-acre  demonstration  area.  Extrapolating,  we  would  expect  about  20,725  anomalies  in  a 
similar  100-acre  area,  with  20,550  clutter. 

•  Two  TOI  were  found  during  the  intrusive  investigation  of  the  4-acre  area,  for  a  rate  of  0.5 
TOI  per  acre.  In  a  100-acre  site  we  predict  about  50  native  TOI. 

•  The  baseline  is  an  EM61  survey  with  0.5-m  line  spacing.  This  would  be  used  to  select 
anomalies  for  digging  without  classification  and  the  same  anomalies  would  be  interrogated 
with  advanced  sensors  and  classified. 

•  The  site  is  seeded  at  a  rate  so  on  average  one  seed  will  be  encountered  each  day  of 
classification  sensor  data  collection.  With  an  estimate  of  20,725  total  anomalies  and  a 
production  rate  of  200  anomalies  per  day,  we  seed  a  conservative  125  inert  items.  These  QC 
seeds  would  be  used  whether  classification  was  used  on  the  site  or  not. 

•  The  classification  performance  is  as  achieved  by  a  number  of  the  analysts  with  ~75%  of  the 
clutter  correctly  identified  and  remaining  undug. 

The  unit  costs  assumed  are  as  shown  in  Table  8-1. 


Table  8-1 .  Unit  cost  assumptions 


Item 

Units 

Cost 

Digs 

Per  hole 

$125 

Classification 

Per  anomaly 

$25  and  $35 

EM61  Survey  Data  Collection  and  Analysis 

100  acres 

$94,000 

Seed  Emplacement 

125  seeds 

$22650 

With  these  assumptions  the  costs  were  calculated  using  the  elements  shown  in  Table  8-2.  A 
traditional  remediation  would  use  125  QC  seeds  and  be  based  on  an  EM61  detection  survey.  Using 
the  numbers  above,  this  effort  would  result  in  the  intmsive  investigation  of  the  125  seeds,  50  UXO, 
and  20,550  clutter  items  and  require  nearly  $2.9M. 

Classification  would  utilize  the  same  number  of  QC  seeds  and,  at  present,  be  based  on  the  same 
EM61  survey.  Far  fewer  items  will  need  intmsive  investigation  using  classification,  only  5138  in  this 
example.  If  classification  can  be  done  for  $35  per  anomaly  including  both  data  collection  and 
analysis,  a  48%  overall  savings  is  possible.  If  this  cost  can  be  lowered  to  $25  to  classify  each 
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anomaly,  which  was  the  cost  for  the  TEMTADS  2x2  team  (Ref.  8),  the  potential  project  savings 
increases  to  55%. 


Table  8-2.  Cost  Comparison  for  100  acres  of  a  site  comparable  to  Camp  Beale 


Item 

No  Classification 

j  Classification 

Quantity 

Cost/$ 

Quantity 

Cost/$ 

($35  /  anomaly) 

Cost/$ 

($25  /  anomaly) 

Seeds 

125  items 

65,600 

125  items 

65,600 

65,600 

EM61  Survey 

1 00  acres 

94,000 

100  acres 

94,000 

94,000 

Classification 

n/a 

0 

20,725 

anomalies 

725,375 

518,125 

Seeds  Dug 

125 

15,625 

125 

15,625 

15,625 

Native  UXO  Dug 

50 

6,250 

50 

6,250 

6,250 

Clutter  Dug 

20,550 

2,756,250 

5138 

642,250 

642,250 

TOTAL 

2,894,175 

1,506,150 

1,298,900 

Percent  Savings 

48% 

55% 

In  this  example,  the  added  cost  of  classification  ranges  from  $520,000  to  $725,000  depending  on  the 
cost  per  anomaly.  This  is  equal  to  the  cost  to  investigate  4000  to  5000  anomalies.  Therefore,  in  this 
example,  the  breakeven  point  is  reached  when  ~25%  of  the  anomalies  are  correctly  classified  as 
clutter.  As  shown  in  Figure  7-10,  all  analysts  in  this  demonstration  exceeded  this  performance. 
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9  CONCLUSIONS 


The  TEMTADS  2X2,  MPV,  and  portable  BUD  all  showed  outstanding  classification  ability.  All 
analysts  working  with  cued  data  from  the  three  advanced  sensors  were  able  to  correctly  identify 
100%  of  the  TOI  while  typically  rejecting  75  to  85%  of  the  clutter.  The  results  among  the  three 
sensors  were  quite  similar.  Figure  7-10  shows  that  nearly  all  of  these  analysts  set  their  thresholds  at  a 
point  that  rejected  all  but  about  250  clutter,  regardless  of  the  source  of  the  data.  The  ability  of 
analysts  to  set  thresholds  correctly  has  been  uneven  in  past  demonstrations:  it  is  now  routine  for 
analysts  to  identify  100%  of  the  TOI  at  their  operating  threshold. 

Using  this  performance,  we  have  constmcted  a  cost  model  for  a  100-acre  remediation  in  conditions 
similar  to  those  of  this  demonstration.  Using  reasonable  assumptions  for  the  cost  of  applying  these 
methods,  we  find  that  the  use  of  classification  would  save  approximately  half  of  the  cost  of  a 
traditional  remediation. 

The  man-portable  sensor  results  are  comparable  to  those  achieved  using  the  MetalMapper  during  a 
concurrent  demonstration  at  the  same  site.  The  targets  at  this  site  were  relatively  shallow  however, 
Figure  5-4,  so  a  final  judgment  on  this  equivalency  will  have  to  await  future  demonstrations. 

These  sensors  are  intended  for  use  on  wooded  sites.  The  Camp  Beale  site  was  relatively  benign  in 
terms  of  access,  but  presented  geolocation  challenges.  Primarily,  this  impacted  the  detection  survey, 
where  less  accurate  geolocation  required  a  change  to  procedures  for  reacquiring  targets  for  cued  data 
collection.  The  procedures  used  in  this  demonstration  were  effective  in  collecting  cued  data  over  all 
anomalies,  confirmed  by  the  ability  to  reacquire,  collect  data  and  correctly  classify  all  the  seeds. 
Additional  demonstrations  will  present  more  challenging  wooded  environments. 

It  is  hard  to  measure  productivity  from  the  first  demonstration  of  these  sensors.  The  field  crews 
were  still  working  out  field  procedures  during  this  demonstration.  Nevertheless,  the  rate  of  the 
most  efficient  data  collectors  was  similar  to  what  is  typically  seen  with  the  MetalMapper.  All  analysts 
requested  more  training  data  for  the  man-portable  sensors  than  for  the  MetalMapper.  This  is 
expected  as  there  is  a  much  smaller  stock  of  existing  library  data  for  these  systems  and  is  likely  to 
change  as  their  field  use  increases. 

As  with  earlier  sites  with  small  munitions  and  relatively  high  anomaly  densities,  the  standard  EM61- 
MK2  data  proved  to  have  limited  classification  value. 

Additional  demonstrations  of  the  portable  sensors  will  address  new  challenges.  The  smallest 
munition  of  interest  to  date  has  been  a  37-mm  projectile.  The  applicability  of  these  techniques  on 
sites  containing  smaller  munitions  and  submunitions  remains  unknown.  Upcoming  demonstrations 
will  continue  to  test  the  performance  of  these  systems  on  sites  with  challenging  terrain,  vegetation 
and  geologic  interference. 
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